ViBERTgrid: A Jointly Trained Multi-modal 2D Document Representation for Key Information Extraction from Documents

Recent grid-based document representations like BERTgrid allow the simultaneous encoding of textual and layout information a in 2D feature map so that state-of-the-art image segmentation and/or object detection models can be straightforwardly leveraged to extract key from documents. However, such methods have not achieved comparable performance sequence- graph-based as LayoutLM PICK yet. In this paper, we propose new multi-modal backbone network by concatenating an intermediate layer CNN model, where input is grid word embeddings, generate more powerful representation, named ViBERTgrid. Unlike BERTgrid, parameters BERT our are trained jointly. Our experimental results demonstrate joint training strategy improves significantly representation ability Consequently, ViBERTgrid-based extraction approach has on real-world datasets.